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Abstract 

The ability of feed-forward neural net archi- 
tectures to learn continuous-valued mappings 
in the presence of noise is demonstrated in 
relation to parameter identification and real- 
time adaptive control applications. Factors 
and parameters influencing the learning per- 
formance of such nets in the presence of noise 
are identified. Their effects are discussed 
through a computer simulation of the Back- 
Error-Propagation algorithm by taking the ex- 
ample of the cart-pole system controlled by a 
non-linear control law. Adequate sampling of 
the state space is found to be essential for can- 
celing the effect of the statistical fluctuations 
and allowing learning to take place. 

1 Introduction. 

A major challenge for health-monitoring and 
diagnosis of complex Advanced Propulsion 
Systems (APS), such as the Space Shuttle 
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Main Engine, is to perform real-time analy- 
sis of a massive amount of diverse sensor data. 
Such analysis can be used either to directly 
perform low-level, real-time adaptive control 
or to send real-time descriptions of the dy- 
namical state of the APS to a high-level con- 
troller. In the first case, the low-level con- 
troller has to compute adaptively and in real- 
time the control signal to be applied to the 
controlled process for a given set of sensor 
data. In the second case, the diagnosis process 
consists of ’’translating” in real-time the sen- 
sor information into one or several parameters 
which characterize specific aspects of the dy- 
namical evolution of the APS. (For a discrete 
mapping, parameter identification reduces to 
pattern recognition). The massive parallelism 
of neural nets [1] and their ability to learn com- 
plex continuous mappings [2] give them the 
real-time computational power needed for such 
low-level monitoring/diagnosis functions. 

When a realistic model of physical phe- 
nomena and feedback mechanisms is possible, 
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adaptive control laws and parameter identi- 
fiers can be expressed in analytical forms, and 
neural nets can be trained from computer- 
generated data. In most instances however, it 
is difficult to derive realistic models, and neu- 
ral nets can only be trained on experimental 
data which have been corrupted by a certain 
amount of noise during the signal processing 
of the sensor measurements. 

In this paper, we analyze the effect of noise 
on the neuromorphic abilities of feed-forward 
neural net architectures to learn continuous 
mappings, and to serve as parameter identi- 
fiers or real-time adaptive controllers . A dis- 
cussion of a computer-simulation of the train- 
ing architecture described in Section 2 is pre- 
sented in Section 3 for a non-linear continuous- 
valued mapping by taking the example of a 
controlled cart-pole system. The training se- 
quence is analyzed in detail with and without 
noise in the training data. 

2 Training Architecture. 

In the presence of noise, the state variables 
of the controlled process, Z, and the applied 


control signal, C, are 


Z = Z exact + h s 

(1) 

and 


C = C exact + h c 

(2) 


where the noises hg and he are simulated 
as independent variables, normally distributed 
with zero mean. For a high signal-to-noise ra- 
tio of the input signal, i.e. rnax , \^\ N >> 1, 

y/var(n g ) 

the function <f> which represents the control 
law, <p(Z exact ) = C exact , can be expressed, us- 
ing a Taylor expansion, as 

+ _ _ (3) 

oZ ( z=z e * act ) 


In this case, trying to learn the mapping <j) 
from sets of noisy input/output vectors (Z, C) 
is thus equivalent to trying to learn <j> from sets 
of input/output vectors (Z exact ,C n ) where 
only the control force is corrupted by noise: 

C n = C exact + h (4) 


- - - / r \ 

n ric — — - (5) 

z dZ(z^z^) 

This equivalence shows that the factors 
and parameters which influence neuromorphic 
learning with noisy input/output data can be 
studied by analyzing the learning performance 
of the net as a function of the signal-to-noise 
ratio m<r ; |cex 1 of the output signal. 

y 'var(n) 

The first phase of the training consists of 
sampling at various times t* the state variables 
Z exact (tk) of the controlled process, Fig.(l), 
and the control signal C n (tk) given in Eqs.(4) 
and (5). In the second phase, the set of train- 
ing data (Z exact (t k ), C n (t k )) is organized in 
I/O subsets before being applied to the neuro- 
morphic controller. 

3 Cart-Pole System. 

In this example, the controlled process of Fig- 
ure 1 is chosen to be the cart-pole system 
[3] - [4] represented in Figure 2. Training 
data are recorded by placing initially the cart- 
pole at arbitrary positions (X(O),0(O)) with 
zero velocities, and by driving it to the ori- 
gin (X = 0,0 = 0) with a control force. 
While the cart-pole is returned to its equi- 
librium position, the four-dimensional state 
vector Z(t) = (X(t), X(t), 0(f), 0(f)), and the 
control force, F(t), are regularly sampled over 
a certain period of time. Sampling rate and 
observation time are considered as parameters 
of the training. 
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Non-Linear Control Law. 

The dynamic evolution of the cart-pole with 
its parameters shown in Figure 2 is given by 
the equations of motion 


before exciting/inhibiting a connected neuron. 
With the standard activation function 

output = — 7 (13) 

r 2 _j_ g-tnput ' ’ 


where 


h 


e = h, -h 2 x 

(6) 

ir f' + F 

x ~ h 

(") 

_ 3 . „ 

h\ — -^gsind 

(8) 

_ 3 

h 2 — —— cosu 
4 L 

(9) 

3 

m(Lsin96 2 — -gsin26) - fX 

(10) 

3 

f 2 — M 4 - m(l — -cos 2 9) 

(11) 


The control force F is generated by applica- 
tion of a feedback linearizing and decoupling 
transform [5] 


F — — [hj 4- k 2 0 -i-ciX -rc 2 X] — fi (12) 


with ki = 25, k 2 = 10, c\ = 1 and c 2 = 
2.6, Training data are generated by integrat- 
ing the set of Eqs.(6)-(12) with the initial 
condition Z( 0) = (A*(0), 0, 0(0), 0), X(0) and 
0(0) being arbitrary position and angle of the 
domain D^^ = [-4m,4*4m] x [-50deg,-j-50degl. 
The neural architecture chosen to approximate 
the control law F[Z] is a feed-forward net of 4 
linear neurons in the input layer, two hidden 
layers of 16 and 4 neurons respectively, and 
one neuron in the output layer. The continu- 
ous values of Z = (X, X ,9,0) are fed into the 
input layer. Each neuron input is connected to 
all neuron outputs of the previous layer, and 
to a unit which is permanently on (threshold 
term). The output signal of each neuron is 
linearly modulated by the (synaptic) weights 


the output o n of a neuron is bound to the inter- 
val [0.1]. Restricting the information domain 
of a neuron to the interval [0.1, 0.9] by elim- 
inating the asymptotes of the input/output 
response curve of Eq.(13) is knowm to en- 
hance learning performance [6]. The neu- 
romorphic learning of the continuous-valued 
mapping F[Z] requires the scaling and offset- 
ting of the last layer output 

F 

u = — = 2.5 *o n - 1.25 (14) 

*9 

where F g is a constant parameter which nor- 
malizes the control signal over [-1,4-1]. It is 
essential to emphasize that the choice of F g 
defines the domain where the mapping is to 
be learned, and influences the neural compu- 
tation as well. Eqs.(13) and (14) show that 
the net output cannot match values of F such 
that |F| > 1.2b F g (practically, the net can 
only match accurately the domain \F\ < F g ). 
When a control law or a continuous-valued 
mapping is to be learned, it is imperative to 
first define the domain of variations of the sig- 
nals before choosing the constant parameter 
F g since it has to be larger than the abso- 
lute maximum value of the signal, max\F \ . 
Among the possible values of F g satisfying 
F g > max \F j , the optimal value corresponds 
to the best approximation of the continuous- 
valued mapping by the neural net. 

It is well-known that linear mappings - those 
for which the control signal is a linear func- 
tion of the state variables - can be learned by 
a single-layer perceptron. In that case, it is 
easy to show that perfect learning occurs over 
any domain of variations of the signals when 
the limit F g — ♦ oo is taken. In the general 
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case however, the only way to find the opti- 
mal F g is to train the net for several values F g 
larger than max\F\, and compare the learning 
performances. In this work, we have chosen 
F g = max\F\ ad hoc. 

The results hereafter have been obtained 
by training the net with the Back- Error- 
Propagation (BEP) algorithm on a VAX 8650 
of the NASA Lewis VAXCLUSTER. The ini- 
tial values of weights and thresholds have been 
chosen randomly distributed in the interval 
[—0.1, +0.1] to break symmetries that could 
eventually lead to spurious modes and bias the 
learning. 

3.1 Absence of Noise. 

In order to extract the relational law between 
state variables and control force from a bulk 
of input /output data, it is necessary that the 
training data be as representative as possible 
of the state space, and that they be sampled 
as uniformly as possible throughout the state 
space. First, if there are too few training data 
or if they are not sufficiently representative, 
the neural net may either only ” memorize’' 
the training data without extracting the con- 
trol law itself, or approximate the control law 
only over a partial domain. This means that 
the net may not be able to reproduce control 
signals corresponding to state variables that 
were not part of the training set. Second, if 
the sampling is not uniform throughout the 
state space, learning may be biased, and the 
net may only ” memorize” part of the training 
data without extracting the features contained 
in the training set. 

When the dynamic characteristics of the 
cart-pole are regularly sampled until it returns 
to the origin, the distribution of the training 
forces {u} is strongly peaked around the ori- 
gin. Clearly, a random sampling of the train- 
ing data would bi&s the training and prevent 
the net from learning the control law for large 


displacements of the cart-pole. Before train- 
ing, data are organized by dividing the inter- 
val [-1,+1] of the normalized force u ~ yr into 
11 subintervals {/*, k = 1, 11} of equal length. 
One training iteration consists of the random 
sampling of a subinterval h followed by the 
random sampling of a u E I*. T c he corre- 
sponding set of state variables is used as input 
to the net. The resulting network output is 
compared to the target output o n = u4 l> l 5 2b ■ 
Each individual error between target output 
and network output is propagated through all 
the net layers to update weights and thresh- 
olds by BEP. This process is iterated until con- 
vergence. 

The accuracy of the neural approximation 
of the control law can be characterized by the 
total mean-square error e 2 


e 2 = 


ii 


(15) 


where is the mean-square error over the 
subinterval I*: 


ek 


2 


srli ) («(o n * t -«(o targ,t ) 3 

n(k) 


(16) 


In Eq.(16), u(i) tar9et is one of the n(k) tar- 
get data of Ifc, and u(i) net the output of the 
net corresponding to the same state variables. 
The convergence of the algorithm can be es- 
timated by evaluating the error e 2 , Eq.(I5), 
made by the net in trying to reproduce the 
data used for training; similarly, the degree of 
accuracy of the feature extraction can be es- 
timated by evaluating the error e 2 made by 
the net in trying to predict the control forces 
corresponding to state vectors that were not 
presented to the net during training. When 
both errors are small and comparable in mag- 
nitude, the neural net has developed a good 
internal representation of the mapping. 

It is well-known that any a-priori knowledge 
on the mapping features, such as symmetries, 
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can efficiently improve the net performance 
when these are incorporated explictly in the 
neural computation. Since the control force 
changes as F — ♦ —F when the state vector 
changes as Z — > —Z, the ensemble of train- 
ing data has been chosen symmetric under the 
transformation T~ — [(Z, F) — > (-Z, -i 7 )] by 
randomly distributing the initial position and 
angle of the cart-pole over the domain Dxe = 
[-4m,-f4m] x [-50deg,-f50deg], 

Since all subintervals {I k } are treated with 
equal probability during training, there need 
to be enough training data to provide as uni- 
form a representation of the control law over 
each Ik as possible. In this section, training 
was performed over 200 motions sampled at 
20Hz over 10s. An upper-bound for the control 
force needed to bring the cart-pole back to the 
origin from a position in Dxe • and without ini- 
tial velocities, is max\F\ = 120Newtons. The 
BEP algorithm was run for F g — 120 N, with 
a steepest descent coefficient rj — 0.2 and a 
momentum term a — 0.9. Since the BEP al- 
gorithm is based on a minimization principle, 
there is always a possibility that the system of 
weights/thresholds may get trapped in a local 
minimum or may freeze in a flat spot during 
the search for the global minimum. In learn- 
ing the mapping F[Z], Eq.(12), and using it 
to control the cart-pole, it was found that the 
neuromorphic controller was able to stabilize 
the pole to 6 = 0, but would occasionally re- 
turn the cart to a position X ^ 0. In order 
to circumvent this difficulty due to the exis- 
tence of a local minimum or a flat spot, we per- 
formed a fine-tuning by augmenting the train- 
ing with data randomly sampled in the smaller 
subinterval I 6 which is centered around 0 and 
symmetric under T~ . 

After training, the learning performance of 
the net can be tested in three different con- 
figurations. The ’’learning open-loop configu- 
ration” tests the ability of retrieving F from 
state variables Z used for training; the net is 


used as an analog memory . The ’’generaliz- 
ing open-loop configuration” tests the ability 
of generating F from state variables not used 
during training; the net is used as a param- 
eter estimator. The ’’generalizing closed-loop 
configuration” tests the ability of stabilizing 
the controlled process for a motion not used 
during training; the net is used as a real-time 
adaptive controller (RTAC). 

The curves of Figures 3-4 show the result 
of the neural computation in the ’’general- 
izing” modes, after 99000 gross-tuning itera- 
tions and 1000 fine-tuning iterations. In the 
set of Figures 3, the initial state vector is 
Z( 0) = (— lm, 0, 45de<7, 0), and in Figures 4, 
Z{ 0) = (3m, 0, — Zbdeg, 0).The neural net is 
able to return, very satisfactorily, the cart-pole 
to the origin from large angles and large dis- 
placements. This shows that, during training, 
the net has developed an internal representa- 
tion which is a very good approximation of 
the mapping defined in Eq.(12). 

With the above set of parameters, estima- 
tions of the error e 2 , Eqs.(15)-(16), are 0.0004 
in the ’’learning open-loop” mode, and 0.0006 
in a ’’generalizing open-loop” mode for a set 
of 200 randomly generated cart-pole motions. 

3.2 Presence of Noise. 

With noise, the normalized values of the con- 
trol forces used as targets are no longer the 
exact values since 

u^ r9et = u exact + n (17) 

However, the mapping F[Z] can be learned 
by BEP from a training set of represen- 
tative data ul? r9et (i). Given the sta- 
tistically averaged square error function 
— — 2 

< [F[Z] 4- hp - G[Z])~ > over the entire state 
space, the BEP yields a function, G(Z ), which 
minimizes the error as shown variationally in 
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Eqs.( 18 )-( 19 ): 

6 < ( F[Z) + h F -G[Z ]) 2 > 

6G[Z] 

G[Z ] = F[Z] ( 19 ) 

In the absence of noise, learning takes place 
only when the net is provided with many rep- 
resentative training data in order to be able 
to reproduce the features rather than ” memo- 
rize” the data. This is even more crucial with 
noise since the target data are not the exact 
values of the force. 

It is essential to choose small values for the 
steepest-descent parameter 77 in order to mini- 
mize the effect of the fluctuations. For a large 
77, samples with big deviation would overcon- 
tribute to the weights adjustments and mis- 
lead the search for the minimum. For a small 
77, the effect of such deviants tend to be bal- 
anced towards average by the contribution of 
the samples with small deviation since those 
occur with a higher probability. From a ge- 
ometric point-of-view, the fluctuations dras- 
tically complicate the topology of the error- 
surfaces by creating more irregularities and 
introducing more possibilities for the net to 
get trapped in local minima or flat spots. 
Small values of 77 favor adiabatic changes of 
the weights towards paths of the energy sur- 
face which correspond to averaged values of 
the training data, i.e. < u t ^ Tget > — < u exact + 
n >=r u exact . In the absence of noise, when 
the target values are the exact values, the role 
of the momentum term is to speed up the con- 
vergence process by amplifying the weight ad- 
justments. In the presence of noise, a momen- 
tum term would amplify the undesired weight 
changes due to highly deviant data, and make 
adiabaticity more difficult to maintain. For 
this reason, the momentum coefficient is cho- 
sen a = 0 . The price to pay for a small 
steepest-descent coefficient and no momentum 


term is of course more iterations to reach con- 
vergence. 

Fluctuations make it also difficult to gener- 
ate representative sets of training data due to 
the ’’contamination” of the subintervals /*. 
Due to noise, training data are likely to lie in 
subintervals that are different from the subin- 
tervals where the exact data are. For a non- 
uniform density distribution of the data used 
for training, this implies that the sampling by 
subintervals as described in Section 3.1 will be 
less uniform as the noise increases. 

If data were obtained by regularly sampling 
cart-pole motions over ’’long” periods of time, 
e.g. 10 sec or more, the exact density distri- 
bution would be well peaked around 0 . Noise 
would cause these data to spread from the cen- 
ter towards the edges of [- 1 ,-hl]. For a given 
noise intensity, the number of data spread out 
of a subinterval would be proportional to the 
number of data contained in that interval. As 
noise increases, data from subintervals located 
around the center, e.g. h, would increasingly 
populate the neighboring subintervals. For low 
signal- to-noise ratio 5 /A 7 = , 3 r and long 

' y-uar(n) 

observation time, the sampling method of Sec- 
tion 3.1 would most probably select noisy data 
originating from exact data located in I 6 > h 
or J7, regardless of the chosen interval. In the 
limit, the fluctuations around the origin would 
be averaged out, and the net would essentially 
learn that Z — 0 is the equilibrium position 
without being able to stabilize the cart-pole. 

Such a ’’contamination” can only be re- 
strained by as much as possible uniformly sam- 
pling the density distribution of the data be- 
fore training. Since the exact values are not 
known, this can be done by shortening the ob- 
servation time of the cart-pole motions and by 
increasing the number of these motions. Over- 
sampling of similar dynamic states, e.g. the 
equilibrium position, would thus be prevented. 

For the simulation, we have assumed that 
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the mapping to be extracted and learned by 
the net is symmetric with respect to the T~ 
transformation, and we have chosen the ini- 
tial positions/angles of the cart-pole randomly 
distributed over Dxe • Towards the end of 
the training, fine-tuning has been obtained 
by training the net with data only sampled 
from the subinterval Iq which is smaller than 
[- 1 ,-flj and still symmetric with respect to the 
T~ transformation. Training data have been 
generated by integrating the equations of mo- 
tion ( 6 )-( 12 ), and adding a gaussian noise of 
zero mean. Samples of noisy cart-pole mo- 
tions used for training are shown in the set 
of Figures oa and 6 a, and the results of the 
computation are shown in Figures 5b-e and 
6 b-e for F g = 120 N with noise-to-signal ratios 
N/S=0.1 and N/S= 0 . 2 . 

For N/S =0.1 (Figs 5b-e), training has been 
made by sampling 1000 cart-pole motions at 
20 Hz over only 5 sec. The BEP parame- 
ters are r\ = 0.02 and a — 0; 490,000 gross- 
tuning/ 10,000 fine-tuning iterations have been 
used. In the ’’learning open-loop” configura- 
tion, e 2 = 0.006, and in a ’“generalizing open- 
loop” configuration, e 2 = 0.006 for a set of 
1000 randomly generated cart-pole motions. 

In the RTAC configuration, the net itself 
controls the evolution of the cart-pole; the 
state vector Z(t) is therefore different from the 
state vector corresponding to the exact con- 
trol law ( 12 ). This explains the discrepancy 
between Figs. 5b and 5c. 

For N/S=0.2 (Figs. 6 b-e), 4000 cart-pole 
motions have been sampled at 20 Hz over 2 
sec only. The BEP parameters are 77 = 
0.01 and a = 0; 950,000 gross-tuning/50,000 
fine-tuning iterations have been used. In 
the ’’learning open-loop” configuration, e 2 = 
0.008, and in a ’’generalizing open-loop” con- 
figuration, e 2 = 0.008 for a set of 4000 ran- 
domly generated cart-pole motions. 

These results show that, in spite of a sig- 
nificant amount of noise, the net is able to 


learn the control law ( 12 ) within sufficient ac- 
curacy to return satisfactorily the cart-pole to 
its equlibrium position. 

In the presence of noise, an error function 
can be introduced to optimize the parameters 
of the computation and estimate the degree 
of accuracy to which the neural mapping ap- 
proximates the exact unknown mapping. It is 
defined from Eqs.(15)-(16), but where u(i) net 
(expected to be closer to the exact value than 
u(i) taT9et is) is one of the n(k) net outputs 
contained in /*, and u(i) taT9et the correspond- 
ing noisy data. In the statistical limit where 
n(k) — ► 00 and for perfect learning, this mean- 
square error would be minimal and equal to 
the variance of the noise. 

4 Conclusion. 

These preliminary results demonstrate that 
neural networks can extract within a good 
aproximation a continuous- valued, non-linear 
mapping between input and output data when 
the training data are corrupted by noise. In 
order to extract the features contained in the 
training data, it is necessary to sample the 
state space as uniformly as possible. In the 
absence of noise, aproximately uniform sam- 
plings can be obtained by randomly sam- 
pling subintervals of the state space. How- 
ever, the presence of noise mixes the popu- 
lations of these subintervals, making it more 
difficult to reach an approximately uniform 
sampling throughout the state space. Limit- 
ing the observation time of each cart-pole mo- 
tion, and increasing the number of such mo- 
tions has been found to significantly reduce 
this contamination effect. It would be interest- 
ing to analyze the possibility of further reduc- 
ing this contamination by pre-processing the 
training data with the help of neuromorphic 
classifiers, such as counter-propagation feed- 
forward nets, before training with BEP. 
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NORMALIZED CONTROL FORCE NORMALIZED CONTROL FORCE 



Figure 1 . - Training architecture for neuromorphic learning with noise. 



Figure 2. - Cart-pole system. The control force F is applied to the cart in the 
presence of friction. 



(a) Generalizing open loop configuration (parameter estimator). 





TIME, sec 

(b) Generalizing closed loop configuration (RTAC); normalized 
control force. 


TIME, sec 

(d) Generalizing dosed loop configuration (RTAC); pendulum 
angle. 


Figure 3. - Non-linear control law without noise. Initial state vector, 2(0) « (-1m, 0, 45 0). Normalization constant, F g =120 Newtons. Back-Error- 

Propagation parameters: a = 0.2 and t \ = 0.9. 
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0 
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EXACT 

AFTER 100 000 TRAINING 
ITERATIONS WITHOUT NOISE 


(a) Generalizing open loop configuration (parameter estimator). 



(c) Generalizing closed loop configuration (RTAC); cart position. 



I I IV1l_ , 30V 

(b) Generalizing closed loop configuration (RTAC); normalized (d) Generalizing closed loop configuration (RTAC); pendulum 

control force. angle. 


Figure 4. - Non-linear control law without noise. Initial state vector. 1(0) - (3m, 0, -35 0). Normalization constant, F g - 120 Newtons Back-Error- 

Propagation parameters: a - 0.2 and ti *0.9. 
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1.0 


EXACT 

EXACT + NOISE 


.2 
0 
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-.4 
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(a) Corrupted control signal. 



(b) Generalizing open loop configuration (parameter estimator). 



TIME, sec 

(c) Generalizing closed loop configuration (RTAC); normalized 
control force. 





(d) Generalizing closed loop configuration (RTAC); cart position. 



TIME, sec 

(e) Generalizing closed loop configuration (RTAC); pendulum angle. 


Figure 5. - Non-linear control law with noise for noise-to-signal ratio, N/S - 0.1 ; Normalization constant, Fg - 120 Newtons; Back-Error- Propagation 
parameters: a * 0.02 and r\ = 0. Neural Net performance tested over noiseless data after training with noise-corrupted data. 
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1.0 


EXACT 

EXACT + NOISE 




(b) Generalizing open loop configuration (parameter estimator). 



(d) Generalizing closed loop configuration (RTAC); cart position. 



TIME, sec 
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Figure 6. - Non-linear control law with noise for nolse-to-signal ratio, N/S * 0.2; Normalization constant, Fg= 120 Newtons; Back-Error-Propagation 
parameters: a * 0.01 and r\ = 0. Neural Net performance tested over noiseless data after training with noise-corrupted data. 
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